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Reference:  Government  Contract  No.  N00014-09-C-0050,  “Enhancing  Simulation-based 

Training  Adversary  Tactics  via  Evolution  (ESTATE)” 

Charles  River  Analytics  Contract  No.  C08098 

Subject:  Contractor’s  Status  Report:  Quarterly  Status  Report  #7 

Reporting  Dates:  6/15/2009  -  9/15/2010 

Dear  Dr.  Hawkins, 

The  following  is  the  Contractor’s  Quarterly  Status  Report  for  the  subject  contract  for  the 
indicated  period.  During  this  reporting  period  we  have  concentrated  on  Task  4:  Develop  Trainee 
Model  Processing. 


1 .  Summary  of  Progress 

1.1  Strategy  Visualization  in  the  MoneyBee  Dataset 

During  our  previous  reporting  period,  our  analyses  indicated  that  under  the  MoneyBee 
experiment,  students  were  learning  through  challenges,  solving  more  difficult  problems  in  less 
time  as  they  gained  experience.  Our  next  step  was  then  to  understand  the  strategies  employed  by 
students  to  create  models  of  different  players  for  future  experimentation. 

1.1.1  Strategy  Visualization  Method 

To  understand  the  strategies  employed,  we  improved  upon  our  graph-based  visualization  of  the 
strategy  space  to  focus  on  perceptual  methods  for  presenting  the  paths  taken  by  players  through 
the  game  state  space.  These  visualizations  are  presented  in  Figure  1  through  Figure  4  in 
Appendix  A. 

Each  of  the  visualizations  presents  a  single  representative  challenge  problem  in  the  MoneyBee 
dataset.  In  this  case,  the  challenge  problem  faced  is  8  coins  that  add  up  to  82  cents.  The  correct 
solution  is  2  of  each  coin:  2  quarters,  2  dimes,  2  nickels,  and  2  pennies.  Each  game  state  is 
represented  by  a  node  with  a  four  digit  number  (QDNP),  signifying  the  number  of  quarters, 
nickels,  dimes,  and  pennies. 

Our  visualization  technique  uses  a  combination  of  node  color,  node  brightness,  and  edge 
thickness  to  perceptually  reveal  elements  of  the  strategy  space.  First,  each  node  in  the  game  state 
graph  is  color  coded.  Green  nodes  indicate  valid  states  on  the  way  to  the  correct  solution  state. 
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Yellow  nodes  have  gone  over  the  number  of  coins  needed,  but  still  are  below  the  target  coin 
value.  Red  nodes  have  violated  both  conditions,  where  the  number  of  coins  and  the  current  coin 
value  you  are  above  the  solution  state.  The  Blue  node  is  the  goal  state.  Second,  each  node’s 
brightness  is  determined  by  how  often  that  node  is  visited  during  play.  Nodes  that  are  visited  less 
are  scaled  darker.  Bright  green  nodes,  therefore,  are  visited  most  often.  Third,  an  edge  is  placed 
between  the  transitions  between  game  states,  either  by  the  player  adding  a  coin  (black 
arrowhead),  removing  a  coin  (gray  arrowhead),  or  resetting  the  game  (arrow  to  0000  node).  The 
width  of  each  edge  is  scaled  based  on  the  frequency  of  how  often  that  transition  occurs. 

Each  visualization  is  constrained  to  players  who  were  faced  with  this  problem  during  a  particular 
session.  For  example,  Figure  1  shows  the  outcome  for  players  who  encountered  this  problem 
during  their  first  session,  or  early  on  in  their  learning  process.  Figure  4  shows  the  outcome  for 
players  who  encounter  this  problem  in  their  fourth  session,  meaning  they  had  encountered  more 
problems  before  this.  Based  on  our  previous  analysis,  we  had  discovered  that  players  do  perform 
better  in  later  sessions  over  earlier  sessions. 

1.1.2  Analysis  of  Strategy  Visualizations 

Several  trends  emerge  when  comparing  the  visualizations  across  sessions.  First,  the  number  of 
game  states  explored  rapidly  decrease,  indicating  that  novice  players  are  inconsistent  among  one 
another  while  expert  players  develop  common  strategies.  Second,  the  number  of  game  states 
visited  with  violations  also  decreases,  indicating  that  expert  players  can  preemptively  or 
reactively  identify  violating  states  and  recover  from  them  gracefully.  Third,  there  is  a  decrease  in 
the  amount  of  backtracking  or  resets.  Finally,  it  is  clear  that  explicit  dominant  strategies  emerge 
early  on  and  grow  stronger  in  later  sessions. 

Now,  let  us  look  at  each  session  individually.  Figure  1  displays  the  results  for  players  who 
encountered  this  challenge  during  their  first  session.  While  the  number  of  states  visited  is  large, 
dominant  paths  emerge.  Specifically,  there  are  four  dominant  paths  that  emerge.  One  dominant 
path  follows  the  path  of  adding  two  of  each  coin  in  sequence,  starting  from  the  largest 
denomination  to  the  smallest  denomination.  In  other  words,  adding  two  quarters,  then  two  dimes, 
then  two  nickels,  and  two  pennies.  Alternatively,  the  other  dominant  paths  begin  follow  the  same 
initial  path  before  diverging.  In  this  case,  one  of  each  coin  is  added  in  sequence,  starting  from  the 
largest  denomination  to  the  smallest  denomination,  arriving  at  the  1111  game  state.  From  here, 
the  path  diverges  equally  into  three  directions.  One  direction  is  to  repeat  this  process,  adding  one 
of  each  coin  in  sequence,  starting  from  largest  denomination  to  the  smallest.  Alternatively, 
another  path  repeats  the  initial  process,  but  adds  from  the  smallest  denomination  to  the  largest 
denomination  in  sequence.  The  final  path  can  adds  one  coin  of  increasing  denominations,  but 
begins  at  the  nickel,  followed  by  dime  and  quarter  before  finishing  with  the  penny.  All  three  of 
these  strategies  can  be  seen  as  variants  of  a  higher-level  strategy,  that  of  adding  one  of  each  coin 
in  a  sequence  (largest  to  smallest  denomination)  followed  by  a  repeat  of  this  process  (largest  to 
smallest  or  smallest  to  largest  denomination). 

In  addition  to  the  two  dominant  strategies,  there  are  other  major  features  of  the  data  for  first 
session  games.  First,  players  move  into  violating  game  states.  Both  in  going  over  the  number  of 
coins  (yellow  nodes)  and  going  over  the  target  total  amount  (red  nodes).  Furthermore, 
backtracking  is  evident  (indicated  by  the  number  of  grey  arrows  moving  up  the  tree),  including 
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full  resets  (grey  arrows  from  advanced  game  states  back  to  the  0000  game  state)  when  violations 
are  identified.  Finally,  some  traces  indicate  circuitous  routes  to  the  goal  state,  using  more  coins 
indicated  than  necessary  and  then  removing  them  to  arrive  at  the  goal  state. 

There  is  noticeable  difference  in  the  results  of  traces  of  second  session  games  of  this  challenge 
when  compared  to  the  first  session  games.  As  illustrated  in  Figure  2,  adding  a  quarter  as  the  first 
move  is  much  more  prevalent.  Additionally,  the  two  of  each  coin  in  decreasing  denomination  is 
the  dominant  strategy.  However,  closer  inspection  does  illustrate  the  alternative  strategy  of  one 
of  each  coin  in  decreasing  denominations,  however,  once  arriving  at  the  1111  game  state,  the 
strategies  split  equally  between  adding  one  of  each  coin  in  decreasing  denominations  (1111  -> 
2111  2211  ->  2221  ->  2222)  or  increasing  denominations  (1111  ->  1 1 12  1 122  ->  1222  -> 

2222).  This  seems  to  indicate  that  more  experienced  players  have  developed  some  common 
strategies  for  attempting  problems.  Additionally,  there  are  fewer  violations,  much  less 
backtracking,  a  limited  number  of  resets,  and  less  circuitous  routes  to  the  goal  state. 

Moving  to  third  session  games,  as  shown  in  Figure  3,  reveals  a  major  shift.  There  is  a  major 
reduction  in  the  number  of  paths  taken.  The  number  of  branches  at  a  given  node  is  often  only 
one,  indicating  that  players  either  (a)  have  a  plan  in  mind  when  making  a  move,  or  (b)  can 
identify  the  next  best  move  at  each  game  state.  The  quarter-first  move  still  dominates  and  the 
“add  two  of  each  coin  in  sequence  from  largest  denomination  to  smallest  denomination”  strategy 
is  noticeable.  The  interesting  feature  of  the  third  session  games  is  that  no  violating  states  are 
visited  nor  is  there  any  backtracking. 

The  fourth  session  games  in  Figure  4  exhibit  a  dramatic  result.  Adding  a  quarter  first  (and 
second)  is  the  only  initial  move.  At  this  point,  the  “add  two  of  each  coin  in  sequence  from  largest 
denomination  to  smallest  denomination”  is  the  dominant  strategy.  However,  we  do  see  some 
deviation  in  some  traces,  backtracking  when  entering  violating  states  or  pressing  forward  and 
removing  the  violation  to  arrive  at  the  game  state.  However,  the  state  space  of  visited  nodes  is 
dramatically  small  and  driven  in  primarily  one  path.  This  indicates  that  experienced  players 
converge  on  a  dominant  strategy. 

This  visualization  method  has  proved  instrumental  to  analyzing  the  strategies  employed  by 
players  with  various  levels  of  expertise.  Our  next  step  would  be  to  analyze  other  challenge 
problems  to  develop  a  common  set  of  strategies  by  players  that  we  can  model  for 
experimentation  purposes.  We  can  then  perform  experiments  using  our  simulated  trainee  models 
to  present  challenges  that  will  push  players  to  adapt  their  strategies. 
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2.  Scheduled  Items 

In  the  next  reporting  period  we  plan  to  address  the  following  items: 


•  Analyze  additional  MoneyBee  challenge  problem  data  using  the  developed  visualization 
technique  to  elicit  common  strategies  for  simulated  trainee  model  development 

•  Design  and  begin  implementation  of  a  simulation  of  1)  trainees  attempting  challenges,  2) 
assess  trainee  skill  and  strategy,  and  3)  challenges  evolving. 

Sincerely, 


Brad  Rosenberg 
Principal  Investigator 
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Appendix  A:  MoneyBee  Visualizations 


Figure  1:  Strategy  Visualization  for  Session  1  Games 
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Figure  2:  Strategy  Visualization  for  Session  2  Games 
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Figure  3:  Strategy  Visualization  for  Session  3  Games 
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Figure  4:  Strategy  Visualization  for  Session  4  Games 
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